iT邦幫忙

2024 iThome 鐵人賽
DAY 11
0
Kubernetes

當 Quarkus 想要騎乘駱駝並用8腳章魚掌控舵手 系列 第 11 篇

當 Quarkus 遇到 HPA

16th鐵人賽 kubernetes quarkus java
265 瀏覽

  • 分享至 

  • xImage
    •  

上一章節已經定義了 HPA 資源，此章節來實作吧! 本實驗是用費氏數列來消耗 CPU

    @GET
    @Path("fib/{number}")
    @Produces(MediaType.APPLICATION_JSON)
    public Response fib(@PathParam("number") String number) {
        var num = Integer.parseInt(number);
        return Response.ok(fib(num)).build();

    }

    public static long fib(int num) {
        if (num <= 1) {
            return num;
        } else {
            return fib(num - 1) + fib(num -2);
        }
    }

本實驗 Resource 配置如下。其基於此指標設定，HPA 控制器會維持擴縮目標中的 Pods 的平均資源利用率在 80%。使用率是 Pod 目前的資源用量與其請求值之間的比值。

...
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

對於上面定義的 CPU/Memory 使用率，只要 metrics.k8s.io API 存在，這些資源度量指標就是可用，且他們不會在不同的 Kubernetes 叢集中改變名稱。可以如下查詢，透過 metric API 進行查詢，也可以透過 PodMetrics。

$ kubectl --kubeconfig ~/.kube/infrlab -n itachi get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/itachi/pods" | jq
{
  "kind": "PodMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metadata": {
        "name": "quarkus-hpa-6bcc89bf6c-nkj6x",
        "namespace": "itachi",
        "creationTimestamp": "2024-05-07T11:20:10Z",
        "labels": {
          "app.kubernetes.io/managed-by": "quarkus",
          "app.kubernetes.io/name": "quarkus-hpa",
          "app.kubernetes.io/version": "1.0.0-SNAPSHOT",
          "pod-template-hash": "6bcc89bf6c"
        }
      },
      "timestamp": "2024-05-07T11:19:51Z",
      "window": "16.299s",
      "containers": [
        {
          "name": "quarkus-hpa",
          "usage": {
            "cpu": "173751n",
            "memory": "54744Ki"
          }
        }
      ]
    }
  ]
}

$ kubectl --kubeconfig ~/.kube/infrlab -n itachi describe PodMetrics quarkus-hpa-6bcc89bf6c-dqdx5
Name:         quarkus-hpa-6bcc89bf6c-dqdx5
Namespace:    itachi
Labels:       app.kubernetes.io/managed-by=quarkus
              app.kubernetes.io/name=quarkus-hpa
              app.kubernetes.io/version=1.0.0-SNAPSHOT
              pod-template-hash=6bcc89bf6c
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  quarkus-hpa
  Usage:
    Cpu:     122095n
    Memory:  53972Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-05-12T08:45:48Z
Timestamp:             2024-05-12T08:45:25Z
Window:                12.719s
Events:                <none>

實戰

如果將 target.type 從 Utilization 變成 AverageValue，同時設定 target.averageValue 而非 target.averageUtilization 的值，就可以將百分比的單位轉換成絕對值。

kubectl --kubeconfig  ~/.kube/infrlab -n itachi apply -f https://raw.githubusercontent.com/CCH0124/O11y-with-quarkus/main/quarkus-hpa/kubernetes/namespace.yaml
kubectl --kubeconfig  ~/.kube/infrlab -n itachi apply -f https://raw.githubusercontent.com/CCH0124/O11y-with-quarkus/main/quarkus-hpa/kubernetes/kubernetes.yml
kubectl --kubeconfig  ~/.kube/infrlab -n itachi apply -f https://raw.githubusercontent.com/CCH0124/O11y-with-quarkus/main/quarkus-hpa/kubernetes/hpa.yaml

執行完後驗證資源有無被建立。

$ kubectl --kubeconfig  ~/.kube/infrlab get all -n itachi
NAME                               READY   STATUS    RESTARTS   AGE
pod/quarkus-hpa-6bcc89bf6c-pmm7v   1/1     Running   0          119m

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/quarkus-hpa   ClusterIP   10.106.88.26   <none>        80/TCP    119m

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/quarkus-hpa   1/1     1            1           119m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/quarkus-hpa-6bcc89bf6c   1         1         1       119m

NAME                                              REFERENCE                TARGETS            MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 14%/80%   1         10        1          118m

在 horizontalpodautoscaler.autoscaling 中 TARGETS 欄位，左側為 CPU、右側為 Memory，CPU 負載當前是 10%，Memory 是 14%。這些值是 REPLICAS 欄位 POD 的平均值，如下驗證。

我們 Pod 資源所定義的資源如下：

...
          resources:
            limits:
              cpu: 100m
              memory: 256Mi
            requests:
              cpu: 10m
              memory: 128Mi
...

從 Metrics Server 獲取的當下使用量如下：

$ kubectl --kubeconfig  ~/.kube/infrlab -n itachi top pods
NAME                           CPU(cores)   MEMORY(bytes)
quarkus-hpa-6bcc89bf6c-ldxfp   1m           18Mi

因此 TARGET 左側 CPU 欄位是 $1\div 10 = 10%$，Memory 是 $18\div 128 = 14%$。

我們透過 kubectl --kubeconfig ~/.kube/infrlab -n itachi get hpa -w 觀察 HPA 資源狀況，可以看到他會每 15 秒進行一次的驗證。

$ kubectl --kubeconfig ~/.kube/infrlab -n itachi get hpa -w
NAME          REFERENCE                TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
quarkus-hpa   Deployment/quarkus-hpa   <unknown>/80%, <unknown>/80%   1         25        0          0s
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 11%/80%               1         25        1          15s
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 11%/80%               1         25        1          30s
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 11%/80%               1         25        1          45s
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 11%/80%               1         25        1          60s
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 12%/80%               1         25        1          75s
...

環境準備好後透過另一個檔案進行 API 端口的大量請求。並觀察 HPA 資源。

$ kubectl --kubeconfig ~/.kube/infrlab -n itachi apply -f kubernetes/curl-test.yaml

當執行該大量請求的 Pod 之後，可以觀察 HPA 資源動向，如下紀錄。

  1. 在 5m31s 前都是使用 scaleUp 第一個策略，每 60 秒增加 2 個 Pod，此時 Pod 為 7
  2. 在 5m31s 後使用 scaleUp 第二個策略，每 60 秒增加 $40%$ 的 Pod，因為 $7*0.4$ 往上取整數是 3，同時大於第一個策略增加 2 個 Pod，加上 selectPolicy 為 Max 因此，選擇增加 3 個 Pod，此時是 10 個 Pod。
  3. 在 6m31s 時使用 scaleUp 第二個策略，此時 $10*0.4$ 往上取整數是 4，因此變成 14 個 Pod
  4. 在 scaleDown 時等待 200 秒的穩定視窗決定是否要縮容，這期間演算法還在計算，確認後就開始進行縮容策略
NAME          REFERENCE                TARGETS                        MINPODS   MAXPODS   REPLICAS   AGE
quarkus-hpa   Deployment/quarkus-hpa   <unknown>/80%, <unknown>/80%   1         25        0          7s
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 43%/80%               1         25        1          15s
quarkus-hpa   Deployment/quarkus-hpa   40%/80%, 43%/80%               1         25        1          2m1s
quarkus-hpa   Deployment/quarkus-hpa   300%/80%, 43%/80%              1         25        1          2m16s
quarkus-hpa   Deployment/quarkus-hpa   710%/80%, 43%/80%              1         25        3          2m31s
quarkus-hpa   Deployment/quarkus-hpa   830%/80%, 26%/80%              1         25        3          2m46s
quarkus-hpa   Deployment/quarkus-hpa   505%/80%, 20%/80%              1         25        3          3m1s
quarkus-hpa   Deployment/quarkus-hpa   340%/80%, 20%/80%              1         25        3          3m16s
quarkus-hpa   Deployment/quarkus-hpa   300%/80%, 21%/80%              1         25        5          3m31s
quarkus-hpa   Deployment/quarkus-hpa   343%/80%, 18%/80%              1         25        5          3m46s
quarkus-hpa   Deployment/quarkus-hpa   326%/80%, 16%/80%              1         25        5          4m1s
quarkus-hpa   Deployment/quarkus-hpa   208%/80%, 16%/80%              1         25        5          4m16s
quarkus-hpa   Deployment/quarkus-hpa   210%/80%, 16%/80%              1         25        7          4m31s
quarkus-hpa   Deployment/quarkus-hpa   210%/80%, 14%/80%              1         25        7          4m46s
quarkus-hpa   Deployment/quarkus-hpa   151%/80%, 14%/80%              1         25        7          5m1s
quarkus-hpa   Deployment/quarkus-hpa   135%/80%, 14%/80%              1         25        7          5m16s
quarkus-hpa   Deployment/quarkus-hpa   151%/80%, 14%/80%              1         25        10         5m31s
quarkus-hpa   Deployment/quarkus-hpa   152%/80%, 14%/80%              1         25        10         5m46s
quarkus-hpa   Deployment/quarkus-hpa   120%/80%, 13%/80%              1         25        10         6m1s
quarkus-hpa   Deployment/quarkus-hpa   110%/80%, 13%/80%              1         25        10         6m16s
quarkus-hpa   Deployment/quarkus-hpa   110%/80%, 13%/80%              1         25        14         6m31s
quarkus-hpa   Deployment/quarkus-hpa   110%/80%, 12%/80%              1         25        14         6m46s
quarkus-hpa   Deployment/quarkus-hpa   92%/80%, 12%/80%               1         25        14         7m2s
quarkus-hpa   Deployment/quarkus-hpa   77%/80%, 12%/80%               1         25        14         7m17s
... 省略(都一樣是 14 個副本)
quarkus-hpa   Deployment/quarkus-hpa   14%/80%, 16%/80%               1         25        14         23m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         23m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         23m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         23m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         24m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         24m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         24m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         25m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         25m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         25m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         25m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         26m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         26m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 16%/80%               1         25        14         26m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 17%/80%               1         25        12         26m 
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 17%/80%               1         25        12         26m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 17%/80%               1         25        12         27m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 17%/80%               1         25        12         27m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 17%/80%               1         25        10         27m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 18%/80%               1         25        10         27m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 18%/80%               1         25        10         28m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 18%/80%               1         25        10         28m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 18%/80%               1         25        8          28m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 18%/80%               1         25        8          28m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 19%/80%               1         25        8          29m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 19%/80%               1         25        8          29m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 20%/80%               1         25        6          29m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 20%/80%               1         25        6          29m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 20%/80%               1         25        6          30m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 20%/80%               1         25        6          30m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 22%/80%               1         25        4          30m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 22%/80%               1         25        4          30m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 22%/80%               1         25        4          31m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 22%/80%               1         25        4          31m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 25%/80%               1         25        3          31m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 25%/80%               1         25        3          31m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          32m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          32m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          32m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          32m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          33m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          33m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          33m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          33m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          34m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          34m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 30%/80%               1         25        2          34m
quarkus-hpa   Deployment/quarkus-hpa   10%/80%, 43%/80%               1         25        1          35m

簡述架構圖如下

https://ithelp.ithome.com.tw/upload/images/20240902/20104688o2D4ZRpKOv.png

結論

優點：

  • 提高資源利用率: HPA 可以確保應用程式始終有足夠的資源來滿足其需求，從而提高資源利用率
  • 降低成本: HPA 可以自動縮減不必要的 Pod，降低基礎設施成本
  • 提高應用程序可用性: HPA 可以自動擴展應用程式以滿足增加的流量，提高應用程式可用性

缺點：

  • 集群可能會耗盡資源: 在叢集新增節點之前，HPA 將無法增加 Pod 數量，此時 Pod 的狀態可能是處於 Pending。需要使用 Cluster Autoscaler (CA) 自動縮放節點。
  • HPA 無法對應意外激增的需求: 新節點可能需要幾分鐘才能加載，因此很難跟上需求的突然變化。在 HPA 擴展過程中，應用程式 Image 過大，拉取時需要更長時間。
  • 應用程式架構須支援分散式: 無法在不同的伺服器之間分配工作負載，Kafka、MongoDB 等。

參考資源

kubernetes | docs | horizontal-pod-autoscaler-v2
kubernetes | docs | horizontal-pod-autoscale


上一篇
認識 Horizontal Pod Autoscaling(HPA)
下一篇
Kubernetes 上的訪問安全控制
系列文
當 Quarkus 想要騎乘駱駝並用8腳章魚掌控舵手 共 31 篇
  1. 27
    Argo CD - Application in any namespace
  2. 28
    Argo CD - notification
  3. 29
    Argo rollouts - blue/green
  4. 30
    Argo CD - Git strategies
  5. 31
    完結
完整目錄
圖片
  直播研討會
圖片
{{ item.channelVendor }} {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

尚未有邦友留言

立即登入留言
iThome鐵人賽
參賽組數
1064 組
團體組數
40 組
累計文章數
22200 篇
完賽人數
602 人
iT+看影片追技術 看影片追技術 看更多
熱門tag 看更多
15th鐵人賽 16th鐵人賽 13th鐵人賽 14th鐵人賽 12th鐵人賽 11th鐵人賽 鐵人賽 2019鐵人賽 javascript 2018鐵人賽 python 2017鐵人賽 windows php c# windows server linux css react vue.js
熱門問題
熱門回答
熱門文章
IT邦幫忙